Markov Decision Processes with Time-Varying Geometric Discounting

Markov decision processes with exponentially representable discounting

We generalize the geometric discount of finite discounted cost Markov Decision Processes to “exponentially representable” discount functions, prove existence of optimal policies which are stationary from some time N onward, and provide an algorithm for their computation. Outside this class, optimal “N-stationary” policies in general do not exist.

Continuous Markov equilibria with quasi-geometric discounting

We prove that the standard quasi-geometric discounting model used in dynamic consumer theory and political economics does not possess continuous Markov perfect equilibria (MPE) if there is a strictly positive lower bound on wealth. We also show that, at points of discontinuity, the decision maker strictly prefers lotteries over the next period’s assets. We then extend the standard model to have...

Continuous time Markov decision processes

In this paper, we consider denumerable state continuous time Markov decision processes with (possibly unbounded) transition and cost rates under average criterion. We present a set of conditions and prove the existence of both average cost optimal stationary policies and a solution of the average optimality equation under the conditions. The results in this paper are applied to an admission con...

Markov decision processes with time-varying discount factors and random horizon

Solving Structured Continuous-Time Markov Decision Processes

We present an approach to solving structured continuous-time Markov decision processes. We approximate the the optimal value function by a compact linear form, resulting in a linear program. The main difficulty arises from the number of constraints that grow exponentially with the number of variables in the system. We exploit the representation of continuous-time Bayesian networks (CTBNs) to de...